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Abstract 

We propose a generalized entropy maximization procedure, which takes into ac- 
count the generalized averaging procedures and information gain definitions under- 
lying the generalized entropies. This novel generalized procedure is then applied to 
Renyi and Tsallis entropies. The generalized entropy maximization procedure for 
Renyi entropies results in the exponential stationary distribution asymptotically for 
q € [0, 1] in contrast to the stationary distribution of the inverse power law obtained 
through the ordinary entropy maximization procedure. Another result of the gen- 
eralized entropy maximization procedure is that one can naturally obtain all the 
possible stationary distributions associated with the Tsallis entropies by employing 
either ordinary or ^-generalized Fourier transforms in the averaging procedure. 
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1 Introduction 



The inverse power law distributions are ubiquitous in nature, emerging in such 
diverse fields as subregion laser cooling [1] , the heartbeat histograms of healthy 
patients [2], plasmas [3], conservative motion in 2-D periodic potentials [4], 
controlled decoherence [5], rheology of steady-state draining foams [6], DNA 
slippage step- length distributions [7], econophysics [8], earthquake models [9], 
to name but a few. 

On the other hand, it is well-known that the stationary distribution obtained 
from the maximization of the Boltzmann-Gibbs (BG) entropy is of exponen- 
tial form. In this sense, it cannot always be used in modeling phenomena 
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exhibiting inverse power law behavior. Therefore, there has been an increas- 
ing interest in the generalized entropies such as Tsallis [10], Renyi [11] and 
Sharma-Mittal (SM) [12] entropies whose stationary solutions are of inverse 
power law form. Although these different entropy measures yield to inverse 
power law stationary distributions, they differ from one another in many as- 
pects. For example, Renyi entropy is additive whereas Tsallis entropy is not. 
Both Tsallis and SM entropies are nonadditive, but the former is one pa- 
rameter generalization whereas the latter is a two-parameter generalization 
of BG entropy. However, one common structure underlying these generalized 
entropies is that all of them are obtained through the joint generalization of 
the averaging procedures and the concept of information gain. For example, 
Renyi entropy preserves the same definition of information gain as BG mea- 
sure, but makes use of a different averaging procedure than the one used for 
BG entropy, namely, exponential averaging procedure. Although this averag- 
ing procedure seems to stem from information-theoretic approaches, it has 
wide range of applications in nonequilibrium statistical physics. For example, 
the free energy difference for arbitrary transformations is nothing but the ex- 
ponential average of total work done during the process as given by Jarzynski 
equality [13]. This equality is the cornerstone of experimental investigations 
in several diverse fields [14]. 

On the other hand, Tsallis measure preserves the same averaging procedure 
as BG measure i.e., linear averaging, but generalizes the concept of informa- 
tion gain by deforming the logarithmic function. The SM entropy benefits 
generalizations of both kind in its structure [15]. 

Although the mathematical structure of these generalized entropies is well un- 
derstood, the entropy maximization procedure (EMP) applied to these gener- 
alized measures does not take the aforementioned structure into account, in 
general. In fact, the choice of constraints mostly relies on trial and error or the 
arguments of suitability in order to obtain a stationary distribution of inverse 
power law. For example, it is emphasized in the literature that Tsallis and 
Renyi entropies are monotonic functions of one another and therefore yield 
the same stationary distribution under the same constraints. However, there 
is no reason to apply the same set of constraints to both, since their mathe- 
matical structure is completely different. At this point, it is worth remark that 
one can obtain a stationary distribution of inverse power law form even by us- 
ing BG entropy with suitably chosen constraints [16]. However, we discard this 
possibility, since one cannot justify the form of constraint necessary for this 
maximization in a reasonable manner [16, 17]. In short, it has been an open 
problem how the constraints for the generalized entropies must be chosen (see 
Ref. [18] for the choice of constraints and the nonadditive formalism). 

In this paper, we propose a new EMP, which takes into account the gen- 
eral mathematical structure of the generalized entropy measures in terms of 
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the underlying definition of the information gain and generalized averaging 
procedure. This novel procedure will be hereafter called generalized entropy 
maximization procedure (GEMP). 

The outline of the paper is as follows. In section II, we review the mathematical 
structure of the Renyi entropy and generalized averaging procedures. The 
application of GEMP to Renyi entropy is presented in Section III. Section IV 
is the application of GEMP to Tsallis entropy. Conclusions are presented in 
Section V. 



2 Generalized averaging procedures and Renyi entropy 



It is well-known that BG entropy is the linear average of the elementary in- 
formation gain log b (l/pi) associated with an event of probability pi i.e., 



where the linear average is defined as 



w 



(^i)lin — y ' Pi%i 



(2) 



W being the total number of configurations of the system. It should be noted 
that from here on we will use the natural base i.e., b = e and denote log e (l/pj) 
as ln(l/pj) without loss of generality. In order to generalize BG entropy, A. 
Renyi considered whether other forms of averaging procedures are possible 
or not. He then adopted the generalized averaging procedure developed by 
Kolmogorov and Nagumo [19, 20]. Kolmogorov and Nagumo independently 
showed that the averaging procedure must be extended to quasi-linear mean 
defined as 



W / -i \ 



(3) 



where / is a strictly monotone continuous and invertible function called Kolmogorov- 
Nagumo function (K-N function). The importance of this extension of the aver- 
aging procedure is understood, since it succeeds the generalization by preserv- 
ing conformity to Kolmogorov axioms of probability as shown by Kolmogorov 
and Nagumo. Renyi then showed that only two possible K-N functions exist 
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if one is restricted to additive measures i.e., for two systems described by two 
independent probability distributions A and B, the entropy measure satisfies 
S(A U B) = S(A) + S(B | A), where the conditional probability S(B | A) is 
defined as S(B | A) = EiPi(A)S(B | A = AJ. The first possible K-N function 
is the linear mean defined in Eq. (2) and reads f(x) = x. The linear mean 
of the information gain results in the BG entropy i.e., Eq. (1). The second 
possibility is the exponential averaging defined by 



f(x) = c^ 1 "^ + c 2 (4) 

where q is a real parameter, and c±, c 2 are two arbitrary constants [11, 15]. 
The arbitrary constants c\ and c 2 can be chosen, without loss of generality, as 
and respectively. The general expression for the exponential average 
of a quantity, using Eqs. (3) and (4), can then be written as 



(^)exp = T3^ ln {^+Y.^ l ~ q)Xl - Wlin) • (5) 

The expression (l)n n is equal to one (more on this later) so that 

(x l ) CX p = T ^ln^^ (1 "^. (6) 

Now, it is not difficult to see that the exponential average of the ordinary 
information gain results in the Renyi entropy [11, 15] i.e., 

sr(ph ( i,i (s)L^" i (? p -) (7) 

where (-) e xp stands for the exponential averaging procedure defined by Eq. (6). 
Since the exponential average becomes the linear average in the q — > 1 limit, 
the Renyi entropy becomes the BG entropy in the same limit, i.e., as q — > 1. 
In other words, the only difference between the Renyi and BG entropies is due 
to the different averaging procedure used although the same definition of the 
information gain ln(l/pj) is preserved in both measures. 

Note that, through Eq. (5), the exponential average of 1 can be written as 

(l)ex P = -r^— In [l + eM(l) lin - (l) lin l . (8) 
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The above equality shows us that once the normalization through linear av- 
eraging procedure is carried out i.e., (l)n n = 1, the normalization through 
exponential averaging procedure i.e., (l) exp = 1 is ensured, and vice versa. 

At this point, it is worth mentioning that if one sets the arbitrary constants 
in Eq. (4) as Ci = 1, c-i = and 1 — q — —ft, then the Jarzynski equality 
AF = -ft In (exp(-/3W)) hD [13] can simply be written as AF = (W) 



3 GEMP and Renyi entropy 

Since the seminal work of Jaynes, the entropy maximization procedure played 
an important role in obtaining the stationary distribution associated with a 
particular entropy measure. For example, the maximization of the BG entropy 
has been carried out by using the following functional 

*bg = U±) ~ a (l) Iin - ft ( £i ) lin (9) 

\ M I lin 

to obtain the concomitant stationary distribution 

Pl = exp(-S BG + ftU-fte t ), (10) 

denoting (£j)ii n by U, as usual. The stationary distribution in Eq. (10) can 
be cast into the form = 6 ^ l , where the partition function is given by 
Z = e 5sG " /3C/ . It should be noted that the linear averages in Eq. (9) are carried 
out using ordinary probability distribution (e.g., not escort distribution), since 
ordinary information gain written in terms of ordinary logarithmic function is 
used in the definition of BG entropy. This fact can be better understood by 
remembering that a function f(x) is completely and uniquely determined by 
its moments when the moments are calculated in terms of the ordinary proba- 
bility distribution and with the help of ordinary Fourier transform. Although 
the Renyi entropy preserves the same definition of ordinary information gain 
(and must therefore be maximized using ordinary probability distribution), 
the averaging procedure is exponential rather than linear. However, the max- 
imization of the Renyi entropy too is done exactly in the same way as in BG 
entropy in the literature [21-29] i.e., 

^=(ln^) - « (l) Iin - P (^> lin , (11) 

\ " % I exp 
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which is completely inconsistent, since the same averaging procedure is not 
used throughout the functional to be maximized. Instead, a consistent maxi- 
mization would require the use of the following functional 



R 



In 



Pi. 



' exp 



(12) 



exp 



The maximization of this consistent functional yields 



$Pi J2p 9 i 



a 



(l - e*" 1 ) - (3 



e (l-q)( £i -U) _ e (q-l)U 



=0 (13) 



where U is consistently calculated through (£j) exp - After a little algebra, one 
obtains the stationary distribution 



Pi 



a 



T,Pj 

1 (1 -e 9 " 1 ) +(3^ 



Ep] 



3 {l-q){ei-U) _ Jq-l)U\ 



Q 



i 

9-1 



(14) 



In order to eliminate the Lagrange multiplier a, we multiply Eq. (13) by pi 
and sum over the index i to obtain 



a 



,(9-1)17 



1 - ei' 



(15) 



By substituting a given by Eq. (15) into Eq. (14), we obtain the stationary 
distribution associated with the Renyi entropy measure as 



Pi 



[l_^(l_ e (H)( £ ,-t/))]H) 



(16) 



with the notation [a]+ = max{0,a}. The parameter (3* is equal to -, and the 

partition function Z q is given by Z q = (J2jP q j) • This stationary distribution 
asymptotically decays as an exponential i.e., p i oc e~ £l for q G [0,1]. This 
result is in complete agreement with the one obtained from the method of 
multinomial coefficients used by Oikonomou [30], and is different than the 
one obtained through ordinary EMP, since the latter obtains a stationary 
distribution of inverse power law form [21-29]. We emphasize again that the 
ordinary maximization of the Renyi entropy is based on the functional in Eq. 
(11) and therefore inconsistent. 
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The analysis of the stationary distribution given by Eq. (16) shows that it 
asymptotically attains a constant value, independent of the microstate energy 
£j, for q > 1. Therefore, the Renyi measure cannot be used as an entropy 
in a thermostatical sense for the region q > 1. This fact has previously been 
agreed upon due to the fact that the Renyi entropy is neither concave nor 
convex in the aforementioned interval. However, a consistent maximization 
of the Renyi measure simply shows the physical reason at the level of the 
stationary distribution (i.e., Eq. (16)), why this measure cannot be used for 
q > 1. The stationary solution obtained through ordinary EMP of the Renyi 
measure, on the other hand, does not show such an inconsistency for the 
interval q > 1. This result also verifies the recent findings of Oikonomou 
and Tirnakli given in [31], where they mentioned that the inverse power law 
distributions obtained from the ordinary EMP for Renyi entropies do not make 
these entropies extensive. Thus, these probability distribution functions do not 
maximize Renyi entropies. Finally, we note that the stationary distribution 
given by Eq. (16) becomes the one given by BG entropy i.e., Eq. (10) in the 
q — > 1 limit. 



4 GEMP and Tsallis entropy 

We will now work in the continuous domain so that the summation of the pre- 
vious section will be replaced by integral. As we have noted before, generalized 
entropies is based on the joint generalization of the concept of information gain 
and the averaging procedure. The Renyi entropy (which is additive like BG 
entropy) preserves the ordinary definition of the information gain, but general- 
izes the averaging procedure. Tsallis entropy (which is nonadditive) is another 
generalized entropy measure, and is based on the generalization of the infor- 
mation gain only, since it preserves the linear averaging procedure of the BG 
entropy. Tsallis entropy reads 




where ln q (x) is q- logarithm given by 

7-1-9 _ i 

\n q (x) = — — . (18) 

In other words, Tsallis entropy preserves the linear averaging procedure, but 
generalizes the definition of the information gain ln(l/p) to the g-information 
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gain denned as ln 9 (l/p). Then, it is evident that, according to GEMP, the 
functional to be maximized must be of the form 




a(-)lin - P{~)lin 



(19) 



where a and (3 are as usual Lagrange multipliers associated with the linear 
average of 1 and the microstate energy e, respectively. Although we know 
that the constraints must be written as linear averaged quantities in accor- 
dance with the fundamental structure of the Tsallis entropy, the form of the 
probability distribution to be used in the functional is not trivial due to the 
generalization of the ordinary definition of the information gain. This is de- 
noted by the brackets with single and double dots. The number of dots in the 
brackets are different, since the same probability distribution may not be used 
in the linear averaging procedure. 

Our goal is now to determine the form of the function f(x) to be used in the 
linear averaging procedure (since Tsallis entropy preserves the linear averaging 
procedure as it is), in Eq. (19). One might here consider three choices: the first 
choice is to employ ordinary definition of the probability function f(x), which 
simply corresponds to ordinary Fourier transform implying all finite moments 
to be calculated through (x n )i in = f dxf(x)x n , (n = 0, 1, ...). This is called first 
choice of constraints in the literature [10,18]. However, the second and third 
choices of constraints are related to g-Fourier transform [32] instead of the 
ordinary one, since a function f(x) can alternatively be determined by its q- 
moments as a result of rewriting the definition of the information gain through 
g-logarithm. Within this scheme, the third choice can easily be obtained as 
follows: the q- Fourier transform [32] of probability density f(x) is given by 



F q [f](0= J ^e ? (^[/(x)rV(*),?>l 



(20) 



— oo 




exp (x) = [1 + (1 - q)x] 



1-q 
+ 




Then, it is possible to show that 
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where q n = 1 + n(q — 1) and g n -mean moments (x n ) qn are given by 



(x n ) - f dxxn f 9n (23) 

with 



v qn = J dx[f(x)r. (24) 

— oo 

This result obtained by Tsallis et al. [32] is very important from our point of 
view, since, as emphasized by Tsallis et al. too, it shows that one must use 
the following generalized escort distributions 



'*<*> = Tsfc- n = - 1 ' 2 '- (25) 

whenever one needs to calculate an averaged quantity in g-space. The results 
of Tsallis et al. [32] can be summarized as follows: the probability density that 
must be used in Eq. (19) is the generalized escort distributions given by Eq. 
(25). If we want to obtain the linear average of a constant, we substitute n = 
in Eq. (25). For any first moment, we substitute n — 1 and so on. Therefore, 
we write, for the linear average of 1 in the g-space, as 

^'K'^'-ffl' 1 (26) 

The inspection of the above equation shows us that the normalization has 
to be carried out in terms of the ordinary probability density function f(x). 
Next, we consider the linear average of the microstate energy e(x) in g-space 

(e)r - fdxf (x)xe- iMlMt - [MlMl (27) 
(£)lm " J dXtqAX) " ~ fdx[f(x)]v - !dx[f{x)Y ■ (27) 

At this point, it should be emphasized that the linear average above is taken 
in terms of the generalized distribution given by Eq. (25) as a result of em- 
ploying g-moments. The maximization of the functional in Eq. (19), subject to 
constraints Eqs. (26) and (27), yields the well-known stationary distribution 



[l-(l-q)P(e-U q )/Jdxp(x)f^ 

P = 7} 128J 
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where U q = (e)u n , and the partition function Z q is given by 



/ 



dx 



l-(l-q)(3 



(g - u«) 

J dxp(x) q 



V(i-?) 



(29) 



Eq. (28) is the well-known probability distribution associated with the third 
choice of constraints [18]. Moreover, the probability distribution associated 
with the second choice of constraints can be obtained from the formalism 
above solely by using unnormalized g-moments. 

Summing up, the algebra underlying the first choice is ordinary Fourier trans- 
form with q', whereas the second and third choices require the use of g-Fourier 
transform with q, satisfying q' — 2 — q [31, 33]. Therefore, all choices used so 
far in nonextensive statistical mechanics [18] emerge naturally within GEMP. 



5 Conclusions 



The emergence of the generalized entropy measures rendered the generaliza- 
tion of the maximization procedure necessary. These generalized entropy mea- 
sures generally stem from the interplay of the generalization of the information 
gain and/or averaging procedure. For instance, the Renyi entropy is obtained 
as a generalization of BG entropy through the generalized averaging proce- 
dure i.e., the exponential average, whereas Tsallis entropy is a generalization 
through a novel definition of information gain. Despite all these generaliza- 
tions of the entropy measures in terms of information gain and/or averaging 
procedures, the maximization procedure has in general been applied without 
taking these changes into account properly. We therefore proposed a general- 
ized entropy maximization procedure, which takes into account the averaging 
procedure and information gain underlying the generalized entropies. This 
novel procedure was then applied to the Renyi and Tsallis entropies. 

Since the Renyi entropy is a generalization of BG entropy in terms of exponen- 
tial averages, the generalized maximization procedure requires the consistent 
use of the exponentially averaged constraints. This novel procedure applied 
to the Renyi entropy then yields a stationary distribution, which asymptot- 
ically decays as an exponential for q e [0,1], instead of inverse power law 
distributions obtained from the ordinary maximization procedure. It should 
be noted that this result is in complete agreement with the one obtained from 
the method of multinomial coefficients [30]. Moreover, the inspection of the 
stationary distribution obtained from the maximization of the Renyi entropy 
shows that it asymptotically attains a constant value, independent of the mi- 
crostate energy for q > 1. Therefore, the Renyi measure cannot be used as 
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a thermodynamical entropy in the region q > 1. This interval was generally 
excluded by recourse to the fact that the Renyi measure is neither concave 
nor convex in the aforementioned interval. The consistent maximization of 
the Renyi measure in this work simply shows the underlying reason why this 
measure cannot be used for the region q > 1 at the level of the stationary dis- 
tribution. The stationary solution obtained through ordinary maximization of 
the Renyi measure, on the other hand, seems to be valid for all q values. 

The nonadditive Tsallis entropy preserves the linear averaging procedure in its 
definition, but deforms the definition of the information gain. As a result, there 
exist two possibilities: the first one is to carry out the averaged constraints 
in terms of moments determined by the ordinary Fourier transform, whereas 
the second possibility is to make use of g-moments, stemming from the q- 
Fourier transform, in the averaging procedure. The former corresponds to the 
probability distribution associated with the first choice of constraints. On the 
other hand, the probability distribution associated with the third choice of 
constraints is obtained from the latter. Finally, the use of unnormalized q- 
moments results in the probability distribution related to the second choice of 
contraints. In other words, in our unifying scheme, all choices of constraints 
naturally emerge depending on the adoption of either ordinary or g-deformed 
Fourier transforms. The resulting stationary distributions, in all cases, are 
genuine inverse power laws. 
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